Sliding Window Attention

The MiMo-V2.5 model family introduces Hybrid Sliding Window Attention (Hybrid SWA) to reduce KVCache storage to roughly 1/7 of Full Attention, sparse MoE activation to cut per-token compute, and multi…

JetBrains released Mellum2, a 12-billion-parameter Mixture-of-Experts model with 2.5 billion active parameters per token, under the Apache 2.0 license. The model is specialized for software engineerin…

DeepSeek-V4's million-token context capability stems from a hybrid attention architecture that compresses context before KV storage, reducing cache pressure. Together's early bring-up on NVIDIA HGX B2…